Information-theoretic lower bounds for distributed statistical estimation with communication constraints
نویسندگان
چکیده
We establish lower bounds on minimax risks for distributed statistical estimation under a communication budget. Such lower bounds reveal the minimum amount of communication required by any procedure to achieve the centralized minimax-optimal rates for statistical estimation. We study two classes of protocols: one in which machines send messages independently, and a second allowing for interactive communication. We establish lower bounds for several problems, including various types of location models, as well as for parameter estimation in regression models.
منابع مشابه
Complexity of Distributed Statistical Algorithms
This paper constructs bounds on the minimax risk under loss functions when statistical estimation is performed in a distributed environment and with communication constraints. We treat this problem using techniques from information theory and communication complexity. In many cases our bounds rely crucially on metric entropy conditions and the classical reduction from estimation to testing. A n...
متن کاملDiscussion on “Minimax Optimal Procedures for Locally Private Estimation”
We congratulate Professors Duchi, Jordan and Wainwright on their path-breaking work in statistical decision theory and privacy. Their extension of classical information-theoretic lower bounds of Le Cam, Fano, and Assouad to local differential privacy can potentially lead to a systematic study of various lower bounds under all kinds of privacy constraints. Their successful treatments of some int...
متن کاملDistributed Nonparametric Regression under Communication Constraints
This paper studies the problem of nonparametric estimation of a smooth function with data distributed across multiple machines. We assume an independent sample from a white noise model is collected at each machine, and an estimator of the underlying true function needs to be constructed at a central machine. We place limits on the number of bits that each machine can use to transmit information...
متن کاملOn Lower Bounds for Statistical Learning Theory
In recent years, tools from information theory have played an increasingly prevalent role in statistical machine learning. In addition to developing efficient, computationally feasible algorithms for analyzing complex datasets, it is of theoretical importance to determine whether such algorithms are “optimal” in the sense that no other algorithm can lead to smaller statistical error. This paper...
متن کاملLocal Privacy, Data Processing Inequalities, and Statistical Minimax Rates
Working under a model of privacy in which data remains private even from the statistician, we study the tradeoff between privacy guarantees and the utility of the resulting statistical estimators. We prove bounds on information-theoretic quantities, including mutual information and Kullback-Leibler divergence, that depend on the privacy guarantees. When combined with standard minimax techniques...
متن کامل